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ABSTRACT 

The objective of this paper is to discuss the 
tttili7:ation of Stake's theory of responsive evaluation by a unit in 
state government charged with the evaluation of the Illinois Migrant 
Program. Through interviews with state and local Title I staffs we 
were able to discover program purposes and concerns and to later 
conceptualize these concerns into issues and problems. Expanding on 
Stake^s modelr our approach includes a larger variety of audiences 
and the use of information from one project to resolve problems in 
another. The report is a dissemination vehicle as well as a federally 
mandated evaluation. (Author) 
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USE OF RESPONSIVE EVALUATION? 



Marjorie Kalman 
Program Review and Documentation Unit 
Department for Exceptional Ciaildren 
Illinois Office of Education 



IN STATEWIDE PROGRAM EVALUATION 




In statewide program evaluations, the most common method used to 
acquire data is through what Stake calls "preordinate*' evaluations* 
By preordinate evaluations, he means evaluations that emphasize 
"I) statement of goals, 2) use of objective tests, 3) standards held 
b/ program personnel, and 4) research-type reports." (Stake, 1974, p*7)* 
Stake suggests that there can be another way to learn how effectively a 
program is being implemented. He calls this method "responsive" 
evaluation. "Responsive evaluation is less reliant on formal communica- 
tion, more reliant on natural communication." (Stake, 1974, p. 7) It 
orients more directly to program activities than to program intents, it 
responds o audience requirements for information, and it refers to the 
different value perspectives of the people at hand in reporting the 
success and failure of the program. 

Although responsive evaluation makes little use of flow charts and 
test scores, it does not ignore objective measures if they are primary 
components of the instructional program. "They are treated not as the 
basis for the evaluation plan, but as components of the instructional 
plan. These components are to be evaluated just as other components 
are. " (Stake, 1974, p. 10, Underlining added.) 



Since objective data can be a component of responsive evaluation, 
why can^t a statewide evaluation be conducted that includes what 
people do naturally when they evaluate things observe and react. 
Why can*t a statewide evaluation orient more directly to program 
activities than to program intents, respond to audience requirements 
for information, and refer to the different value-perspectives of the 
people at hand in reporting the success and failure of the program? 

These questions have become the concern of the Program Review 
and Documentation Unit (P.R.D.U.) of the Illinois Office of Education, 
an internal unit within the Department for Exceptional Children, This 
unites responsibilities are to evaluate special education programs 
in Illinois that are state or federally funded. The five members of the 
unit spend most of their time evaluating the Elementary and Secondary 
Act Title I programs, which include programs for the educationally 
disadvantaged child, the migrant child, and those neglected/delinquent 
or handicapped children who are served by state operated or state 
supported facilitieSo 

Until 1972, the federally mandated evaluations of the Title I 
programs were contracted to universities, thus preventing continuous 
year round follow-up of the Title I programs. In the fall of 1972, 
the P.R.D.U. was given responsibility for the annual evaluations; the 
unit was not commissioned, it was assigned. Over the past two-and-a- 
half years, this unit has evolved from one that has reported numerical 
statistical data to one that is more concerned with issues that may 
have an effect on the quality of program services. 

The P.R.D.U. has at least three audiences to which it responds when 
evaluations are written. One audience is the federal government, 
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which is a mandated audience. To meet the needs of this audience, 
preordinate evaluation data is required. Another audience or client 
is the State Education Agency, especially the coordinators of the 
Title I programs. This paper, however, will be focusing on the migrant 
program. The migrant coordinator has been an acl * re client, providing 
input to the P.R.D.U* regarding the needs and concerns of the migrant 
program that he feels should be examined, A third client is the 
migrant project directors. They are passive clients, providing little 
input in the form of comments or criticisms to the P.R.D.U, 

Althoujph the evaluation of the migrant program is compiled because 
it is the law, the unit has chosen to not only report numbers, but to 
point out areas of concern that could affect the state and local 
administrators of the migrant projects. By doing this, the P.R.D.U. 
hopes to educate its audience as to the potential of evaluation reports. 

In order to most effectively discuss how the unit has attempted 
to employ Stake's method of responsive evaluation in evaluating the 
migrant program. Stake's list of 12 recurring events (see Figure 1), 
which he has diagrammed in the form of a clock face will be used. 
Stake emphasizes that any event can follow any event, and that obser- 
vation and feedback must be continuous. This clock moves clockwise, 
counter-clockwise, and cross-ciockwise. Many' of the events can occur 
simultaneously and the evaluator may return to each event several times 
during the course of the evaluation. 

Since the migrant program evaluator had written two annual reports, 
she already had an overview of the propram activities . She then 
approached the migrant coordinator in order to conceptualize issues or 
problems . The migrant coordinator made an assumption that successful 
migrant projects had changed and were continuing to chanpe community 
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attitudes towards the migr^^nts. He felt this was especially trt e of 
the eight week summer migrant projects. He wanted the unit to observe 
and record the techniques used by the successful projects so that other 
sites could use this information to chan^ee their local community 
attitudes. Observers were selected and instruments were developed . 
Two members of the P.R.D.IJ, visited three summer migrant project sites 
*^or twu days each and interviewed staff, parents, to^^ispeople, and 
children, not only to learn about the project/child relationship, but 
to also determine the community/migrant relationship and the effect of 
the school program on this relationship, • 

Through talking ^^th program staff , the P.R.D.U, learned that the 
assumption that successful projects had changed community attitudes 
towards the migrants was invalid. Any effect of the schcol programs 
on the community/migrant relationship was not discernable to the 
observers. The migrant project tended to operate in isolation from the 
local community. Those teachers and staff that were interested in 
working with migrant children formed a temporary communitv that shared 
a genuine concern for the future of the migrant children. 

The members of the P,R,D,U, were required to be flexible and change 
the focus of the visitations, Overviewing program activities at the 
local level, identifying program scope, discovering purposes and jon* 
cerns, and conceptualizing issues and problems throuph talks with the 
staff at the migrant sites became the goal of the visits. 

Descriptions of the three sites were written up and included in the 
formal report , the fiscal year 1974 evaluation. These portrayals 
described each project as seen by the staff, parents, townspeople, 
and children interviewed. Below is one portrayal. 
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Site: Princeville Elementary School 
Princeville, Illinois ^ 

Congressional District: H8 

Individuals Interviewed: Project Director 1 Aides 2 

Recruiter 1 older Children 6 

Teachers 6 Parents A 

Townspeople ^^ 

Princeville is a rural community with a population of 1,455 located 
in central Illinois. The Joan of Arc canning company 'has a plant there, 
and hires migrants from Texas to pick crops such as asparagus, peas, 
corn and pumpkins between May and October of each year. Housing in camps 
is provided for the migrants, rent free, during their stay in Princeville. 
The migrant summer school project has been operating for eight years. 

The summer school program was much more informal than a typical 
regular term program. The children were free to move about the classrooms; 
swimming was part of the curriculum; field trips were taken once a week. 
To encourage older students to attend school, parents were informed that 
the children could arrive at school at any time; tardiness held no conse- 
quences in summer school. The children were welcome to come to school 
after they had worked in the fields. Career awareness classes, where 
students were exposed to a variety of jobs that were realistically 
possible for them, were provided as incentives for the older childrea. 

Two summer school programs operated concurrently at the Princeville 
Elementary school this past summer. One program was a remedial program 
funded through ESEA Title I 89-10 funds that served the children who 
lived year round in Princeville. The other program was the ESEA Title I 
•89-750 migrant program. Last year, the 89-10 summer school classes were 
combined with the migrant classes. This year, the school board voted 
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against the integrated summer school program, because they felt the town 
children should be receiving strictly remedial work rather than academics 
combined with field trips and swimming • The Princeville migrant school 
staff hope to reinstate an integrated summer program next year^ 

Project Director 

Ms, N, the project director, had been teaching in the migrant summer 
school program for the past five years; Ms, N was brought up in the area 
around Princeville, although she taught elsewhere during the academic 
year. This was her first year as project director, 

Ms. N and the director of all Title I programs in the district 
wrote the proposal for the project. Staff were selected based on their 
interest in teaching migrant children and their qualifications. Since 
eight Princeville teachers were already teaching in the 89-10 summer 
school program, only two summer teachers were from Princeville* 

Ms. N saw the strength of the project in "staff flexibility and 
their great attitude toward migrant kids." She saw, for herself and her 
teachers > a need for long range planning. Ms. N felt that she could 
improve as an administrator by determining specific goals whereby her 
staff could best serve the migrant children in an eight week program. 
She would like to see the teachers develop reasonal le classroom goals 
that could be accomplished in th<» time allotted, rather than skim oyer 
material in order to meet an unreasonable expectation, Ms. N would like 
to organize ^. work study program for next summer, but she did not know 
the best methods for implementing such a program or how to approach 
potential employers, 

Ms, N evaluated her staff through classroom observation. She was 
always in the building and was easily accessible to all teaching and 
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non-teaching staff. She felt she needed as3lstance in developing 
different and more sophisticated evaluation Lcols, but Ms. N did not 
know who to turn to for assistance. 

Recruiter 

The recruiter for the Princeville program, Mr. G, was a sociology 
student from San Antonio, Texas. His experiences during the summer were 
to be incorporated in a paper for which he would get field work credit. 
He had been living in the camp for the past two months witli the knowledge 
of the camp foreman. By living in the camp, Mr. G was able to get closer 
to the migrant parents and earn their trust. 

Mr» G had varied responsibilities. He had visited every dwelling in 
the three camps in the area and had obtained the names, ages, and schools 
last attended for all children under 16. He took care of attendance 
records and sent updated migrant student record transfer system forms to 
Springfield. Mr. G took children to clinics and translated prescriptions 
for the parents. He also tutored children who had little or no knowledge 
of English. 

Mr. G felt a need for greater interaction between the migrants and 
the community. He suggested the organizing of a welcoming committee. 

Teachers 

This was Ms. H's third summer at the project. She is not bilingual 
and she teaches in the northern part of the state during the school year. 
She is "in love with the migrant kidsw" She is very impressed with the 
closeknit family life of the migrant and described them as the hardest 
working people she has met. 

In class, she wanted to improve the migrant child's abilities An 
reading, language and math. She wanted "to make them fit into the world 
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t • . to build their self -concept." Ms. H has had good experiences with 
the migrant parents she has worked with. She has found them to be very 
interested and supportive, commenting that she would rather work with 
these parents than her regular term Anglo parents. 

Ms. D was teaching for the very first time. She found that her 
picture of the migrant child was idealistic. The children were less 
excited about learning than she expected. Her goals for the children 
were: 1) to improve skills they had already been taught, 2) to instill 
the idea that learning can be fun and challenging, but a challenge that 
they can meet, 3) to be aware of other job opportunities, and A) to be 
happy with themselves and considerate of others. Her long term goal was 
to eliminate the children from the migrant stream. 

Ms. B, a bilingual candidate for a Ph.D. in Spanish, was teaching ' 
at Princeville for the first time. She taught high school students in 
Peoria during the regular term. Although she was interviewed in 
February, she was hired just before the summer session since another 
teacher turned down .the job. Consequently, she did not have an oppor- 
tunity to plan her eight week curriculum. She used several methods for 
identifying individual strengths and weaknesses in the areas of math, 
reading, and writing of both Spanish and English. She usually divided 
the children into three ability groups. She and the aide would each work 
with a group while the third group worked independently. When there were ^ 
no field trips, class would start with P.E., followed by one-and-a-half hour 
of English /Language Arts and then one-and-a-half hours of math and some 
science, utilizing math games as much as possible. After lunch she worked 
on writing and spelling in Spanish, art and music. She used no planned 
curriculum; she "winged it" on a day to day basis. As a consequence, she 
had no idea how to evaluate the program, 
O 9 
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Ms, B, Felt the state office could help migrant teachers by supplying 
"teaching English as a second language BiaterialsJ' 

Mr. W, the Texas teacher, had spent several summers in Princeville. 
He has a strong desire to help migrant children get out of the migrant 
stream through education, although he had commented during a later conver- 
sation that "there will always be more migrants because as the present 
ones gain education or other skills and drop out, new migrants from 
Mexico will come," 

He noticed that many of the children view the summer school as a 
vacation school. He felt the majority of the parents liked the summer 
school because: 1) it kept the children out of the camps, 2) it served 
as a day care center, 3) it was a good educational experience, and 4) it 
was a vacation school. He had heard directly that some parents were 
concerned that the children were not getting enough academics. 

Mr. W felt a need for more academic tools. He suggested that a hand- 
book be developed locally for each grade. This handbook would include 
general lesson plans to help the teachers. He would like to see greater' 
use of behavioral objectives and the development of assessment tech- 
niques to better diagnose the childrens' weaknesses and teach those skill 
areas. 

Ms. D teaches in a neighboring community daring the regular school 
year, and has been teaching in the Princeville summer program for the past 
three years. Ms. D's classroom goals were: 1) to improve the migrant 
child's self -concept, 2) to make the child bilingual since these children 
are usually poor in Spanish and English, and 3) to keep them In school. 
She saw the ultimate goal of the program to make the migrants see the value 
of education. She noted that the children are starting school earlier and 
staying in longer. Ms. D commented that the migrants do not perceive 
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their situation to be as bad as Anglos do. They see their life as hard 
work and hard play. 

Ms. D, like the other teachers here, visited the parents in the 
camps approximately once a week. She noted that, in the past, parents 
did not send their children to summer school o Parents were now more 
supportive, but if money was tight work would come first. 

Ms. D. felt the communitjr was afraid of the migrants because they 
didn't understand them. She noted that in spring of 1973, the former 
project director held a meeting for the Anglo parents whose children would 
be in the integrated program and explained some of the cultural differences 
between the Anglos and tfhe migrants. Most of the parents who attended the 
meeting sent their children to summer school. Ms. D found that a more 
positive attitude change occurred in both groups of children as a result 
of the integrated program. Ms. D was sorry the integrated program was 
discontinued. 

This was a^so Ms. S*s third summer teaching at Princeville. She 
loved working with migrant children and felt there was a positiye atti- 
tude among the summer school teachers. Ms. S*s goals were: 1) to make 
the children like school, 2) to make them like themselves, and 3) to. 
provide some academics but to emphasixe self -concept . Her long term goal 
was to see them leave ths migrant stream. 

Ms. S thought the attitude of the migrant and the Anglo children 
had improved during the integrated class sessions last summer. 

Aides 

Ms. T, a migrant aide, has been coming to Princeville for the past 
eight years and has been an aide for six. She was certified with the 
suraner program in Illinois as well as the program in Texas. She was 
excited that her daughter was learning good Spanish in the Texas school 




and was visiting new places and getting remedial help in the Prtnceville 
program. She also commented that the summer school program kept the 
children busy while the parents were working. Ms. T liked the idea of 
teachers visiting the homes. Although she liked the Texas and summer 
school teachers, she felt the regular term Princeville teachers were too 
strict with the Anglo and the migrant children. 

Ms. L, another migrant aide, has been coming to x'rinceville for ten 
years and has been an aide for six. Her responsibilities appeared to be 
general: aiding the children with math and reading, helping them write 
their names and numbers, watching the class when the teacher left the 
room. She felt the program was good and that the children learned a lot. 
She commented that her daughter was learning more in Illinois than in 
Texas, although she felt that the summer term Princeville teachers 
responded more warmly to the migrant children than did the regular term 
teachers. Ms. L felt home visits were helpful in some cases, and she had 
no suggestions for improving the school program. 

Older Children 

Manuel was 13 years old and had been coming to Princeville for two 
years. He liked the school and his teachers. His favorite subjects 
were art, math and gym and he especially liked the field trips. He was 
satisfied with the segregated classrooms, since this arrangement allowed 
him to play with all his friends from the camp, although he stated that 
he also played with children from the town. 

Donna, lA, had been coming to Princeville for "a long time." She 
liked school, especially the field trips and the vacation classes (sewing, 
cooking and shop). Donna preferred the segregated grouping since there 
were so many fights last year. She commented, ''My parents send me to 
school because there is nothing important I can do in the camp." 
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Two other older students who were Interviewed had been coming to 
Princeville for three years. They especially enjoyed the field trips 
and swimming. One child liked the integrated program last year, while 
the other child preferred being with the migrant children. 

Five town children from the summer Title I program were interviewed. 
All had participated in the integrated program the previous year. Of the 
first two children interviewed, one was an Anglo and ona was a settled 
out migrant. The Anglo child preferred summer school this year because 
the day was shorter and he felt he was learning more. He did miss the 
field trips. The settled out child preferred the mixed class to segre- 
gated classes, while the Anglo child didn't care. Neither had heard any 
preferences by tVeir parents. 

Parents 

Three families were interviewed at the camps. All three families 
welcomed the interviewers courteously, including a ready invitation to come 
inside their clean and orderly, but cramped for space, cabins. 

In the first cabin Mr. and Mrs. S welcomed us into their one room 
which they shared with their child. Mrs. S had been coming to Princeville 
every summer since she was born 21 years ago. She was enrolled in the 
school system as a child and could compare her experiences with those of 
her child. The addition of a summer project, via Title I 89-750, had 
made quite a difference. She especially liked the fact that the teachers 
came out to the camps to talk with parents about their child's progress. 
Mr. and Mrs. S were so concerned about their child's education that they 
were going to stay in Illinois this year so their child could begin 
kindergarten (in Illinois a child can start school if he is five by 
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December 1). They would put. their child in a private school if they 
went back to Texas although the cost would be high. 

Their child liked school very much. Mrs. S did not care for how the 
migrant children were grouped and segregated when they came in April. The 
school district claimed it did this to avoid large classrooms** 

Another parent felt the program at the school was excellent. She 
liked the field trips and was glad the summer program was not too 
academically oriented. 

A third parent, Mr. Y, was in Princeville for the first time. He 
was a construction worker in Texas. His older sons were working in the 
factory because Mr. Y felt the school was not providing enough academics. 
However, the boys were to attend school in the fall. The younger son, 
perhaps reflecting the impressions in the home, did not like summer 
school because he was not learning enough. Mr. Y had suggested that the 
boy thinks of the school as a "vacation school". The father probably 
utilized the school as a free day care service. 

Townspeople 

Since Princeville was a very rural community. It was possible to * 
interview townspeople regarding their impressions of the migrant summer 
school program. 

The bank aicecutive^s perception of the migrant summer program was 
that it was a day camp. He was opposed to ^-he number of field trips 
and felt that summer school was a time for remedial work. He was in 
favor of the concept of a summer school program for migrants, but he 
felt the emphasis was in the wrong place. 

* The migrant coordinator has noted that migrant children were pulled out 
of regular classrooms for an 89-750 funded tutorial program only. 
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child. He was concerned abouc the" needs of his own children. He 
mentioned that his daughter needed help in math last year, but that she 
could not get that help unless a tutor was hired. 

The druggist had heard of the program at thp. summer school. Mr. P 
had no strong feelings about the school, but he stated that some to\ms- 
people were jealous because they felt the migrant children were getting 
extra privileges at the summer school. The druggist* s son-in-law, a 
member of the town council, thought the money was well spent. 

Mr. S had an extensive historical perspective of the migrant 
influence on the community over the past 27 years. He was currently an 
administrator at the Joan of Arc plant in Princeville, but he had worked 
as a recruiter for migrant help in Texas as a foreman. 

Until his heart attack last winter, he had been serving on local 
school boards for 20 years. In this time, he had noticed a big difference 
in the education of migrants. The school "... attaches them to our 
society." Mr. S felt the children needed the continued structure that 
summer school gave them, and he noted that the "school keeps children 
busy if they can*t work." He felt the school could benefit the migrants 
by enabling them to get out of the stream. 

When asked about the integrated summer school program that was held 
last summer, Mr. S noted that it had some good social and fiscal (econom- 
ically more efficient) benefits. However, due to the vociferous demands 
of several parents, the board voted to have separate but equal programs. 
Mr. S was on the borderline but he left early and did not vote. He 
said. 
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It is probably better this way, because the to\m wasn't 
ready yet for an integrated program, ir.aybe in another 
five years. For example, when you brought these tv/o 
groups together last year the slow Anglos were mixed in 
v;ith migrant students who ranged from slow to very 
bright. Because of this uneven mix, the migrant kid 
will always v;in. 

School was viewed as competition betwer^n groups to Mr. S. He had difficulty 
accepting the idea that an Anglo could perform less effectively than a 
migrant. This feeling was shared by others, since hard feelings developed 
when a migrant won the sewing contest last year. 

Regarding summer school, Mr. S. felt, '*The field trip route may bring 
more kids into the school, but time would more wisely be spent on basics. 
This will develop better leaders who' can help their people." 
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As a result of the visits to the migrant projects, the P.R.D.U. 
was able to conceptualize ^'^^ues and problems regarding program 
effectiveness that were not reflected in the portrayals. Instead, the 
coordinator of the migrant program was interviewed and these concerns 
were discussed in the interview. At this point, the P.R.D.U. progressed 
from recording happenings and values as perceived by project staff to 
assuming an interventionist role. Based on the site visits, the P.R.D.U. 
had concerns about the effectiveness of the eight week summer 
projects. Most teachers interviewed had never seen the project 
proposal and had failed to develop long and short range goals for the 
eight weeks. They were unable to specify what changes they expected 
to see in the children as a result of attending summer school or how 
they were going to measure those changes. This concern of limited 
or vague objectives was shared by one of the Princeville teachers who 
was an ex-migrant. He specifically was concerned with the lack of 
lesson plans, curriculum goals, and evaluation tools. 

As a result of the interview with the migrant coordinator 
regarding these findings, the coordinator and his staff decided to 
devote more time to training administrators in conducting needs 
assessments and writing adequate performance objectives. Two training 
sessions have already been held, with a member of the P.R.D.U. attending 
as a participant observer. At these sessions, the local migrant 
directors have been requested to have each teacher of migrant children 
read the proposal before he/she signs a contract. The teacher then 
not only knows what is expected of him/her during the eight week 
summer session, but he/she will also be expected to develop check- 
lists for gathering base line data and assessing pupil progress. 
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The fiscal year 1975 migrant evaluation will include visits to 
six migrant summer projects. Tlie P.R.D.U. will ascertain the 
effectiveness of the administrator training on project implementation 
by observing designated antecedents » transactions and outcomes . 
Instruments will be developed to interview teachers as to their 
classroom objectives and the unit will also be examining the checklists 
developed by the teachers. 

Secondary areas of focus have also developed since the fall of 
1974. The assistant to the migrant coordinator has been developing a 
supplementary phonics program that can be incorporated into any reading 
program. Several sites will be field testing this program and data 
will be kept on control and experimental groups* The results of this 
program will be incorporated in the annual migrant evaluation report, 
thus providing feedback to the sites as to the utility of this phonics 
program. 

Another focus will be the ef f ecti.veness of a special preschool 
intervention program on a migrant population that will be attending 
the program only eight weeks. Again, a sample of migrant projects 
will participate in the preschool program, and the results will be 
included in the annual report. 

The P.R.D.U. represents the state, and it must be careful not to 
alienate the migrant project directors because they conduct the summer 
programs voluntarily. While the unit cannot be too interventionist at 
the local level, it can be at the state level. The migrant coordinator 
has taken past recommendations by the P.R.D.U. and displayed leadership 
by responding . to those recommendations with program change. 
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Stake^s method of responsive evaluation can be applied at the 
state level. State administrators can take the information gleaned 
from interviewing local staff in their natural environment and make 
broad changes that improve the quality of the programs statewide. 
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